{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JPVBc5ndpFIX"
   },
   "source": [
    "# TextAttack & AllenNLP \n",
    "\n",
    "This is an example of testing adversarial attacks from TextAttack on pretrained models provided by AllenNLP. \n",
    "\n",
    "In a few lines of code, we load a sentiment analysis model trained on the Stanford Sentiment Treebank and configure it with a TextAttack model wrapper. Then, we initialize the TextBugger attack and run the attack on a few samples from the SST-2 train set.\n",
    "\n",
    "For more information on AllenNLP pre-trained models: https://docs.allennlp.org/v1.0.0rc3/tutorials/getting_started/using_pretrained_models/\n",
    "\n",
    "For more information about the TextBugger attack: https://arxiv.org/abs/1812.05271"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "AyPMGcz0qLfK"
   },
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/QData/TextAttack/blob/master/docs/2notebook/Example_2_allennlp.ipynb)\n",
    "\n",
    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/QData/TextAttack/blob/master/docs/2notebook/Example_2_allennlp.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gNhZmYq-ek-2"
   },
   "outputs": [],
   "source": [
    "!pip install allennlp allennlp_models textattack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "RzOEn-6Shfxu"
   },
   "outputs": [],
   "source": [
    "!pip install datasets pyarrow transformers --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_br6Xvsif9SA"
   },
   "outputs": [],
   "source": [
    "from allennlp.predictors import Predictor\n",
    "import allennlp_models.classification\n",
    "\n",
    "import textattack\n",
    "\n",
    "class AllenNLPModel(textattack.models.wrappers.ModelWrapper):\n",
    "    def __init__(self):\n",
    "        self.predictor = Predictor.from_path(\"https://storage.googleapis.com/allennlp-public-models/basic_stanford_sentiment_treebank-2020.06.09.tar.gz\")\n",
    "\n",
    "    def __call__(self, text_input_list):\n",
    "        outputs = []\n",
    "        for text_input in text_input_list:\n",
    "            outputs.append(self.predictor.predict(sentence=text_input))\n",
    "        # For each output, outputs['logits'] contains the logits where\n",
    "        # index 0 corresponds to the positive and index 1 corresponds \n",
    "        # to the negative score. We reverse the outputs (by reverse slicing,\n",
    "        # [::-1]) so that negative comes first and positive comes second.\n",
    "        return [output['logits'][::-1] for output in outputs]\n",
    "\n",
    "model_wrapper = AllenNLPModel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "id": "_vt74Gd2hqA6",
    "outputId": "c317d64d-9499-449a-ef93-f28be0c0d7a2"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[34;1mtextattack\u001b[0m: Loading \u001b[94mnlp\u001b[0m dataset \u001b[94mglue\u001b[0m, subset \u001b[94msst2\u001b[0m, split \u001b[94mtrain\u001b[0m.\n",
      "\u001b[34;1mtextattack\u001b[0m: Unknown if model of class <class '__main__.AllenNLPModel'> compatible with goal function <class 'textattack.goal_functions.classification.untargeted_classification.UntargetedClassification'>.\n",
      "/usr/local/lib/python3.6/dist-packages/textattack/constraints/semantics/sentence_encoders/sentence_encoder.py:149: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).\n",
      "  embeddings[len(transformed_texts) :]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Result 0:\n",
      "\u001b[91mNegative (95%)\u001b[0m --> \u001b[92mPositive (93%)\u001b[0m\n",
      "\n",
      "\u001b[91mhide\u001b[0m new secretions from the parental units \n",
      "\n",
      "\u001b[92mconcealing\u001b[0m new secretions from the parental units \n",
      "\n",
      "\n",
      "\n",
      "Result 1:\n",
      "\u001b[91mNegative (96%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "contains no wit , only labored gags \n",
      "\n",
      "\n",
      "\n",
      "Result 2:\n",
      "\u001b[92mPositive (100%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "that loves its characters and communicates something rather beautiful about human nature \n",
      "\n",
      "\n",
      "\n",
      "Result 3:\n",
      "\u001b[92mPositive (82%)\u001b[0m --> \u001b[37m[SKIPPED]\u001b[0m\n",
      "\n",
      "remains utterly satisfied to remain the same throughout \n",
      "\n",
      "\n",
      "\n",
      "Result 4:\n",
      "\u001b[91mNegative (98%)\u001b[0m --> \u001b[92mPositive (52%)\u001b[0m\n",
      "\n",
      "on the \u001b[91mworst\u001b[0m \u001b[91mrevenge-of-the-nerds\u001b[0m clichés the filmmakers could \u001b[91mdredge\u001b[0m up \n",
      "\n",
      "on the \u001b[92mpire\u001b[0m \u001b[92mrеvenge-of-the-nerds\u001b[0m clichés the filmmakers could \u001b[92mdragging\u001b[0m up \n",
      "\n",
      "\n",
      "\n",
      "Result 5:\n",
      "\u001b[91mNegative (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "that 's far too tragic to merit such superficial treatment \n",
      "\n",
      "\n",
      "\n",
      "Result 6:\n",
      "\u001b[92mPositive (98%)\u001b[0m --> \u001b[91mNegative (50%)\u001b[0m\n",
      "\n",
      "demonstrates that the \u001b[92mdirector\u001b[0m of such \u001b[92mhollywood\u001b[0m blockbusters as patriot \u001b[92mgames\u001b[0m can still turn out a \u001b[92msmall\u001b[0m , personal \u001b[92mfilm\u001b[0m with an \u001b[92memotional\u001b[0m \u001b[92mwallop\u001b[0m . \n",
      "\n",
      "demonstrates that the \u001b[91mdirectors\u001b[0m of such \u001b[91mtinseltown\u001b[0m blockbusters as patriot \u001b[91mgame\u001b[0m can still turn out a \u001b[91mtiny\u001b[0m , personal \u001b[91mmovie\u001b[0m with an \u001b[91msentimental\u001b[0m \u001b[91mbatting\u001b[0m . \n",
      "\n",
      "\n",
      "\n",
      "Result 7:\n",
      "\u001b[92mPositive (90%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "of saucy \n",
      "\n",
      "\n",
      "\n",
      "Result 8:\n",
      "\u001b[91mNegative (99%)\u001b[0m --> \u001b[92mPositive (83%)\u001b[0m\n",
      "\n",
      "a \u001b[91mdepressed\u001b[0m \u001b[91mfifteen-year-old\u001b[0m 's suicidal poetry \n",
      "\n",
      "a \u001b[92mdepr\u001b[0m \u001b[92messed\u001b[0m \u001b[92mfifteeny-ear-old\u001b[0m 's suicidal poetry \n",
      "\n",
      "\n",
      "\n",
      "Result 9:\n",
      "\u001b[92mPositive (79%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "are more deeply thought through than in most ` right-thinking ' films \n",
      "\n",
      "\n",
      "\n",
      "Result 10:\n",
      "\u001b[91mNegative (97%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "goes to absurd lengths \n",
      "\n",
      "\n",
      "\n",
      "Result 11:\n",
      "\u001b[91mNegative (94%)\u001b[0m --> \u001b[92mPositive (51%)\u001b[0m\n",
      "\n",
      "for those \u001b[91mmoviegoers\u001b[0m who \u001b[91mcomplain\u001b[0m that ` they do \u001b[91mn't\u001b[0m make movies like they used to anymore \n",
      "\n",
      "for those \u001b[92mmovieg\u001b[0m \u001b[92moers\u001b[0m who \u001b[92mcompl\u001b[0m \u001b[92main\u001b[0m that ` they do \u001b[92mnt\u001b[0m make movies like they used to anymore \n",
      "\n",
      "\n",
      "\n",
      "Result 12:\n",
      "\u001b[91mNegative (92%)\u001b[0m --> \u001b[92mPositive (85%)\u001b[0m\n",
      "\n",
      "the part where \u001b[91mnothing\u001b[0m 's happening , \n",
      "\n",
      "the part where \u001b[92mnothin\u001b[0m 's happening , \n",
      "\n",
      "\n",
      "\n",
      "Result 13:\n",
      "\u001b[91mNegative (97%)\u001b[0m --> \u001b[92mPositive (90%)\u001b[0m\n",
      "\n",
      "saw how \u001b[91mbad\u001b[0m this movie was \n",
      "\n",
      "saw how \u001b[92minclement\u001b[0m this movie was \n",
      "\n",
      "\n",
      "\n",
      "Result 14:\n",
      "\u001b[91mNegative (73%)\u001b[0m --> \u001b[92mPositive (84%)\u001b[0m\n",
      "\n",
      "lend some dignity to a \u001b[91mdumb\u001b[0m story \n",
      "\n",
      "lend some dignity to a \u001b[92mdaft\u001b[0m story \n",
      "\n",
      "\n",
      "\n",
      "Result 15:\n",
      "\u001b[92mPositive (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "the greatest musicians \n",
      "\n",
      "\n",
      "\n",
      "Result 16:\n",
      "\u001b[91mNegative (98%)\u001b[0m --> \u001b[92mPositive (99%)\u001b[0m\n",
      "\n",
      "\u001b[91mcold\u001b[0m movie \n",
      "\n",
      "\u001b[92mcolder\u001b[0m movie \n",
      "\n",
      "\n",
      "\n",
      "Result 17:\n",
      "\u001b[92mPositive (87%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "with his usual intelligence and subtlety \n",
      "\n",
      "\n",
      "\n",
      "Result 18:\n",
      "\u001b[91mNegative (99%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "redundant concept \n",
      "\n",
      "\n",
      "\n",
      "Result 19:\n",
      "\u001b[92mPositive (93%)\u001b[0m --> \u001b[91m[FAILED]\u001b[0m\n",
      "\n",
      "swimming is above all about a young woman 's face , and by casting an actress whose face projects that woman 's doubts and yearnings , it succeeds . \n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from textattack.datasets import HuggingFaceDataset\n",
    "from textattack.attack_recipes import TextBuggerLi2018\n",
    "\n",
    "dataset = HuggingFaceDataset(\"glue\", \"sst2\", \"train\")\n",
    "attack = TextBuggerLi2018(model_wrapper)\n",
    "\n",
    "results = list(attack.attack_dataset(dataset, indices=range(20)))\n",
    "for idx, result in enumerate(results):\n",
    "    print(f'Result {idx}:')\n",
    "    print(result.__str__(color_method='ansi'))\n",
    "    print('\\n')\n",
    "print()"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "[TextAttack] Model Example: AllenNLP",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
